Occam's Two Razors: the Sharp and the Blunt Occam's Two Razors Theoretical Arguments for the Second Razor the Pac-learning Argument

Occam's razor has been the subject of much controversy. This paper argues that this is partly because it has been interpreted in two quite different ways, the rst of which (simplicity is a goal in itself) is essentially correct, while the second (simplicity leads to greater accuracy) is not. The paper reviews the large variety of theoretical arguments and empirical evidence for and against the \second razor," and concludes that the balance is strongly against it. In particular, it builds on the case of (Schaaer, 1993) and (Webb, 1996) by considering additional theoretical arguments and recent empirical evidence that the second razor fails in most domains. A version of the rst razor more appropriate to KDD is proposed, and we argue that continuing to apply the second razor risks causing signiicant opportunities to be missed. William of Occam's famous razor states that \Nun-quam ponenda est pluralitas sin necesitate," which, approximately translated, means \Entities should not be multiplied beyond necessity" (Tornay 1938). It was born in the late Middle Ages as a criticism of scholastic philosophy, whose theories grew ever more elaborate without any corresponding improvement in predictive power. In the intervening centuries it has come to be seen as one of the fundamental tenets of modern science , and today it is often invoked by learning theorists and KDD practitioners as a justiication for preferring simpler models over more complex ones. However, formulating Occam's razor in KDD terms it trickier than might appear at rst. Leaving aside for the moment the question of how to measure simplicity, let generalization error of a model be its error rate on unseen examples, and training-set error be its error on the examples it was learned from. Then the formulation that is perhaps closest to Occam's original intent is: First razor: Given two models with the same generalization error, the simpler one should be preferred because simplicity is desirable in itself. On the other hand, within KDD Occam's razor is often used in a quite diierent sense, that can be stated as: Second razor: Given two models with the same training-set error, the simpler one should be preferred because it is likely to have lower generalization error. We believe that it is important to distinguish clearly between these two versions of Occam's razor. The rst one is largely uncontroversial, while the second one, taken literally, is false. Several theoretical arguments and pieces of empirical …